
Human Behavior Understanding

Behavior and properties of spatio%2Dtemporal local features under visual transformations.

Sonify your face facial expressions for sound generation.

The use of non%2Dconventional methods for content analysis and understanding panel overview.

Pervasive video analysis workshop overview.

Human%2Dcentered multimedia systems tutorial overview.

Toward an automatically generated soundtrack from low%2Dlevel cross%2Dmodal correlations for automotive scenarios.

Putting the pieces together multimodal analysis of social attention in meetings.



An Eye Fixation Database for Saliency Detection in Images.

Challenges of Human Behavior Understanding.
Abstract. Recent advances in pattern recognition has allowed computer scientists and psychologists to jointly address automatic analysis of of human behavior via computers. The Workshop on Human Behavior Understanding at the International Conference on Pattern Recognition explores a number of different aspects and open questions in this field, and demonstrates the multi-disciplinary nature of this research area. In this brief summary, we give an overview of the Workshop and discuss the main research challenges 4. 1
Employing social gaze and speaking activity for automatic determination of the Extraversion trait.

Visual Gaze Estimation by Joint Head and Eye Information.
In this paper, we present an unconstrained visual gaze estimation system. The proposed method extracts the visual field of view of a person looking at a target scene in order to estimate the approximate location of interest (visual gaze). The novelty of the system is the joint use of head pose and eye location information to fine tune the visual gaze estimated by the head pose only, so that the system can be used in multiple scenarios. The improvements obtained by the proposed approach are validated using the Boston University head pose dataset, on which the standard deviation of the joint visual gaze estimation improved by 61.06 % horizontally and 52.23 % vertically with respect to the gaze estimation obtained by the head pose only. A user study shows the potential of the proposed system. 1
Best of Automatic Face and Gesture Recognition 2008.

Best of Automatic Face and Gesture Recognition 2008.

Large%2Dscale image and video search Challenges%2C technologies%2C and trends.

Exploiting facial expressions for affective video summarisation.

Webcam%2DBased Visual Gaze Estimation.
Abstract. In this paper we combine a state of the art eye center locator and a new eye corner locator into a system which estimates the visual gaze of a user in a controlled environment (e.g. sitting in front of a screen). In order to reduce to a minimum the computational costs, the eye corner locator is built upon the same technology of the eye center locator, tweaked for the specific task. If high mapping precision is not a priority of the application, we claim that the system can achieve acceptable accuracy without the requirements of additional dedicated hardware. We believe that this could bring new gaze based methodologies for human-computer interactions into the mainstream. 1
Isocentric color saliency in images.

Human%2Dcentered Computing Application to Multimedia.

Proceedings of the 3rd ACM Workshop on Human%2DCentered Computing

3rd international workshop on human%2Dcentered computing %28HCC %2708%29.

Facial expression recognition as a creative interface.
We present an audiovisual creativity tool that automatically recognizes facial expressions in real time, producing sounds in combination with images. The facial expression recognition component detects and tracks a face and outputs a feature vector of motions of specific locations in the face. The feature vector is used as input to a Bayesian network which classifies facial expressions into several categories (e.g., angry, disgusted, happy, etc.). The classification results are used along with the feature vector to generate a combination of sounds and images that change in real time depending on the person’s facial expressions. We explain the basic components of our tool and several possible applications in the arts (performance, installation) and medical domains.
Similarity Matching in Computer Vision and Multimedia.
Comparing two images, or an image and a model, is the fundamental operation for any retrieval systems. The similarity matching of two images can reside in the hierarchical levels from pixel-by-pixel level, feature space level, object level, and semantic level. In most systems of interest, a simple pixel-by-pixel comparison will not suffice: the difference that we determine must bear some correlation with the perceptual difference of the two images or with the difference between two adequate semantics associated to the two images. Similarity matching techniques are developed mostly for recognition of objects under several conditions of the distortion while similarity measures, on the other hand, are used in applications like image databases. Matching and dissimilarity measurement are not seldom based on the same techniques, but they differ in emphasis and applications.
Distance Learning for Similarity Estimation.
Abstract—In this paper, we present a general guideline to find a better distance measure for similarity estimation based on statistical analysis of distribution models and distance functions. A new set of distance measures are derived from the harmonic distance, the geometric distance, and their generalized variants according to the Maximum Likelihood theory. These measures can provide a more accurate feature model than the classical euclidean and Manhattan distances. We also find that the feature elements are often from heterogeneous sources that may have different influence on similarity estimation. Therefore, the assumption of single isotropic distribution model is often inappropriate. To alleviate this problem, we use a boosted distance measure framework that finds multiple distance measures, which fit the distribution of selected feature elements best for accurate similarity estimation. The new distance measures for similarity estimation are tested on two applications: stereo matching and motion tracking in video sequences. The performance of boosted distance measure is further evaluated on several benchmark data sets from the UCI repository and two image retrieval applications. In all the experiments, robust results are obtained based on the proposed methods.
Special section from the ACM multimedia conference 2007.

Proceedings of the 15th International Conference on Multimedia 2007

Proceedings of the 6th ACM International Conference on Image and Video Retrieval
Proceedings of the 6th ACM international conference on Image
Multimedia Content Analysis and Mining
www.infotech.monash.edu
Human%2DComputer Interaction
www.bcs-hci.org.uk Find out what happened at HCI2004 Interacting with … music aeroplanes petrol pumps Published by the British HCI Group • ISSN 1351-119X 1
Human%2Dcentered multimedia systems tutorial overview.

Overview of the ImageCLEF 2007 Object Retrieval Task.
Abstract. We describe the object retrieval task of ImageCLEF 2007, give an overview of the methods of the participating groups, and present and discuss the results. The task was based on the widely used PASCAL object recognition data to train object recognition methods and on the IAPR TC-12 benchmark dataset from which images of objects of the ten different classes bicycles, buses, cars, motorbikes, cats, cows, dogs, horses, sheep, and persons had to be retrieved. Seven international groups participated using a wide variety of methods. The results of the evaluation show that the task was very challenging and that different methods for relevance assessment can have a strong influence on the results of an evaluation. 1
Integrating Relevance Feedback in Boosting for Content%2DBased Image Retrieval.

Two%2DDimensional Adaptive Discriminant Analysis.

Human%2DComputer Intelligent Interaction A Survey.

Human%2DCentered Computing Challenges and Perspectives.

Cooperative Object Tracking with Multiple PTZ Cameras.

Do Colour Interest Points Improve Image Retrieval%3F
In image retrieval scenarios, many methods use interest point detection at an early stage to find regions in which descriptors are calculated. Finding salient locations in image data is crucial for these tasks. Observing that most current methods use only the luminance information of the images, we investigate the use of colour information in interest point detection. A way to use multi-channel information in the Harris corner detector is explored and different colour spaces are evaluated. To determine the characteristic scale of an interest point, a new colour scale selection method is presented. We show that using colour information and boosting salient colours results in improved performance in retrieval tasks.
Personalized multimedia retrieval the new trend%3F
The aim of this paper is to present an overview of the current situation in this hot topic of Multimedia Information Retrieval: Personalization. We are considering several aspects of this problem. On one hand, the user will want to have a personalized access to his image/video collections and this can be achieved by providing intuitive and natural browsing capabilities and customized features. Furthermore, the system is required to perform user profiling and to adapt the existing parameters of the system to the user needs and interest. On the other hand, it is also important to consider the devices and applications in which this technology is going to be deployed. Mobile media is a high growth area but the state-of-the-art technologies are lagging behind the consumers expectations. We are addressing in this paper precisely these three important aspects.
Multimodal human%2Dcomputer interaction A survey.
Abstract. In this paper we review the major approaches to Multimodal Human Computer Interaction, giving an overview of the field from a computer vision perspective. In particular, we focus on body, gesture, gaze, and affective interaction (facial expression recognition and emotion in audio). We discuss user and task modeling, and multimodal fusion, highlighting challenges, open issues, and emerging applications for Multimodal Human Computer Interaction (MMHCI) research.
Guest Editors%27 Introduction Human%2DCentered Computing%2D%2DToward a Human Revolution.

Context%2DBased Object%2DClass Recognition and Retrieval by Generalized Correlograms.
Abstract—We present a novel approach for retrieval of object categories based on a novel type of image representation: the Generalized Correlogram (GC). In our image representation, the object is described as a constellation of GCs, where each one encodes information about some local part and the spatial relations from this part to others (that is, the part’s context). We show how such a representation can be used with fast procedures that learn the object category with weak supervision and efficiently match the model of the object against large collections of images. In the learning stage, we show that, by integrating our representation with Boosting, the system is able to obtain a compact model that is represented by very few features, where each feature conveys key properties about the object’s parts and their spatial arrangement. In the matching step, we propose direct procedures that exploit our representation for efficiently considering spatial coherence between the matching of local parts. Combined with an appropriate data organization such as Inverted Files, we show that thousands of images can be evaluated efficiently. The framework has been applied to different standard databases, and we show that our results are favorably compared against state-of-the-art methods in both computational cost and accuracy. Index Terms—Object recognition, retrieval, Boosting, spatial pattern, contextual information. 1
The age of human computer interaction.
This paper presents a set of evidence-based guidelines for designing websites for the elderly. Extensive literature review was conducted, guidelines were developed and supported by literature, and finally the guidelines were used to evaluate ten websites that have as their primary target audience the elderly. Problems Addressed There is a lot of past research on the topics of aging and human computer interaction by the elderly but no central collection of this information is presented in an organised way. Through this study, a structured classification of usability guidelines was created to address the area of human-computer interaction for the elderly. When information is researched about web design, such as the best way to convey information on a website, the best way to structure the information, how to attract the visitors’ attention and so on, it is common to find that these issues are addressed with a so called typical audience in mind: the more frequent users of the world wide web. One can go so far as to say that the frequent users of the World Wide Web are seen as the young generation by default. The World Wide Web is a profitable business that capitalises on its main asset: information. People visit websites for information retrieval. The purpose is defeated if people visit a website only to be unable to obtain the required information. Sometimes the information is actually there but hidden. Website designers have many techniques at their disposal to attract attention to content on websites. Use of colour, images, and animation are just some of the techniques that can be used. This seems logical and plausible. But the issue is that websites should be accessible by all people. This includes not only the young generation but also those with disabilities and the elderly. One can find guidelines that can be followed to make the experience of information retrieval on websites userfriendly, but these appear as generic. It is feasible to use these generic guidelines to make websites more accessible for the elderly but they need to be more specific and be traceable to the needs of the elderly. It is important to
Authentic facial expression analysis.
It is argued that for the computer to be able to interact with humans, it needs to have the communication skills of humans. One of these skills is the ability to understand the emotional state of the person. The most expressive way humans display emotions is through facial expressions. In most facial expression systems and databases, the emotion data was collected by asking the subjects to perform a series of facial expressions. However, these directed or deliberate facial action tasks typically differ in appearance and timing from the authentic facial expressions induced through events in the normal environment of the subject. In this paper, we present our effort in creating an authentic facial expression database based on spontaneous emotions derived from the environment. Furthermore, we test and compare a wide range of classifiers from the machine learning literature that can be used for facial expression classification.
Computer Vision in Human%2DComputer Interaction
We present a nonintrusive system based on computer vision for human-computer interaction in three--dimensional (3-D) environments controlled by hand pointing gestures. Users are allowed to walk around in a room and manipulate information displayed on its walls by using their own hands as pointing devices. Once captured and tracked in real-time using stereo vision, hand pointing gestures are remapped onto the current point of interest, thus reproducing in an advanced interaction scenario the &#034;drag and click&#034; behavior of traditional mice. The system, called PointAt (patent pending), enjoys a careful modeling of both user and optical subsystem, and visual algorithms for self-calibration and adaptation to both user peculiarities and environmental changes. The concluding sections provide an insight into system characteristics, performance, and relevance for real applications.
Human%2Dcentered computing a multimedia perspective.

Corner Detectors for Affine Invariant Salient Regions Is Color Important%3F.

Toward Robust Distance Metric Analysis for Similarity Estimation.
In this paper, we present a general guideline to establish the relation between a distribution model and its corresponding similarity estimation. A rich set of distance metrics, such as harmonic distance and geometric distance, is derived according to Maximum Likelihood theory. These metrics can provide a more accurate feature model than the conventional Euclidean distance (SSD) and Manhattan distance (SAD). Because the feature elements are from heterogeneous sources and may have different influence on similarity estimation, the assumption of single isotropic distribution model is often inappropriate. We propose a novel boosted distance metric that not only finds the best distance metric that fits the distribution of the underlying elements but also selects the most important feature elements with respect to similarity. We experiment with different distance metrics for similarity estimation and compute
A New Study on Distance Metrics as Similarity Measurement.
Distance metric is widely used in similarity estimation. In this paper we find that the most popular Euclidean and Manhattan distance may not be suitable for all data distribution. A general guideline to establish the relation between a distribution model and its corresponding similarity estimation is proposed. Based on Maximum Likelihood theory, we propose new distance metrics, such as harmonic distance and geometric distance. Because the feature elements may be from heterogeneous sources and usually have different influence on similarity estimation, it is inappropriate to model the distribution as isotropic. We propose a novel boosted distance metric that not only finds the best distance metric that fits the distribution of the underlying elements but also selects the most important feature elements with respect to similarity. The boosted distance metric is tested on fifteen benchmark data sets from the UCI repository and two image retrieval applications. In all the experiments, robust results are obtained based on the proposed methods. 1.
Emotion Recognition Based on Joint Visual and Audio Cues.

The Role of Featural and Configural Information in Face Classification A Simulation of the Expertise Hypothesis.

Boosting the distance estimation Application to the K%2DNearest Neighbor Classifier.

Content%2Dbased multimedia information retrieval State of the art and challenges.
Extending beyond the boundaries of science, art, and culture, content-based multimedia information retrieval provides new paradigms and methods for searching through the myriad variety of media all over the world. This survey reviews 100+ recent articles on content-based multimedia information retrieval and discusses their role in current research directions which include browsing and search paradigms, user studies, affective computing, learning, semantic queries, new features and media types, high performance indexing, and evaluation techniques. Based on the current state of the art, we discuss the major challenges for the future.
Computer Vision in Human%2DComputer Interaction
We present a nonintrusive system based on computer vision for human-computer interaction in three--dimensional (3-D) environments controlled by hand pointing gestures. Users are allowed to walk around in a room and manipulate information displayed on its walls by using their own hands as pointing devices. Once captured and tracked in real-time using stereo vision, hand pointing gestures are remapped onto the current point of interest, thus reproducing in an advanced interaction scenario the &#034;drag and click&#034; behavior of traditional mice. The system, called PointAt (patent pending), enjoys a careful modeling of both user and optical subsystem, and visual algorithms for self-calibration and adaptation to both user peculiarities and environmental changes. The concluding sections provide an insight into system characteristics, performance, and relevance for real applications.
Affective multimodal human%2Dcomputer interaction.
This paper describes an intelligent system that we developed to support affective multimodal human-computer interaction (AMM-HCI) where the user’s actions and emotions are modeled and then used to adapt the HCI and support the user in his or her activity. The proposed system, which we named Gaze-X, is based on sensing and interpretation of the human part of the computer’s context, known as W5+ (who, where, what, when, why, how). It integrates a number of natural human communicative modalities including speech, eye gaze direction, face and facial expression, and a number of standard HCI modalities like keystrokes, mouse movements, and active software identification, which, in turn, are fed into processes that provide decision making and adapt the HCI to support the user in his or her activity according to his or her preferences. To attain a system that can be educated, that can improve its knowledge and decision making through experience, we use case-based reasoning as the inference engine of Gaze-X. The utilized case base is a dynamic, incrementally self-organizing event-content-addressable memory that allows fact retrieval and evaluation of encountered events based upon the user preferences and the generalizations formed from prior input. To support concepts of concurrency, modularity/scalability, persistency, and mobility, Gaze-X has been built as an agent-based system where different agents are responsible for different parts of the processing. A usability study conducted in an office scenario with a number of users indicates that Gaze-X is perceived as effective, easy to use, useful, and affectively qualitative.
Fast Spatial Pattern Discovery Integrating Boosting with Constellations of Contextual Descriptors.
We present a novel approach for fast object class recognition incorporating contextual information into boosting. The object is represented as a constellation of generalized correlograms that integrate both information of local parts and their spatial relations. Incorporating the spatial relations into our constellation of descriptors, we show that an exhaustive search for the best matching can be avoided. Combining the contextual descriptors with boosting, the system simultaneously learns the information that characterize each part of the object along with their characteristic mutual spatial relations. The proposed framework includes a matching step between homologous parts in the training set, and learning the spatial pattern after matching. In the matching part two approaches are provided: a supervised algorithm and an unsupervised one. Our results are favorably compared against state-of-the-art results. 1
Multimodal Human Computer Interaction A Survey.
Abstract. In this paper we review the major approaches to Multimodal Human Computer Interaction, giving an overview of the field from a computer vision perspective. In particular, we focus on body, gesture, gaze, and affective interaction (facial expression recognition and emotion in audio). We discuss user and task modeling, and multimodal fusion, highlighting challenges, open issues, and emerging applications for Multimodal Human Computer Interaction (MMHCI) research.
Neighborhood issue in single%2Dframe image super%2Dresolution.

Video Object Boundary Reconstruction by 2%2DPass Voting.

Affective Meeting Video Analysis.
In this paper we examine the affective content of meeting videos. First we asked five subjects to manually label three meeting videos using continuous response measurement (continuous-scale labeling in real-time) for energy and valence (the two dimensions of the human affect space). Then we automatically extracted audio-visual features to characterize the affective content of the videos. We compare the results of manual labeling and low-level automatic audiovisual feature extraction. Our analysis yields promising results, which suggest that affective meeting video analysis can lead to very interesting observations useful for automatic indexing. 1.
Efficient Object%2DClass Recognition by Boosting Contextual Information.

How to Complete Performance Graphs in Content%2DBased Image Retrieval Add Generality and Normalize Scope.

Systems and architectures for multimedia information retrieval.

Learning probabilistic classifiers for human%2Dcomputer interaction applications.

Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval

Computer Vision in Human%2DComputer Interaction
We present a nonintrusive system based on computer vision for human-computer interaction in three--dimensional (3-D) environments controlled by hand pointing gestures. Users are allowed to walk around in a room and manipulate information displayed on its walls by using their own hands as pointing devices. Once captured and tracked in real-time using stereo vision, hand pointing gestures are remapped onto the current point of interest, thus reproducing in an advanced interaction scenario the &#034;drag and click&#034; behavior of traditional mice. The system, called PointAt (patent pending), enjoys a careful modeling of both user and optical subsystem, and visual algorithms for self-calibration and adaptation to both user peculiarities and environmental changes. The concluding sections provide an insight into system characteristics, performance, and relevance for real applications.
The State%2Dof%2Dthe%2DArt in Human%2DComputer Interaction.
unication that establishes a framework for multimodal &#034;language&#034; and &#034;dialog&#034;, much like the framework we have evolved for spoken exchange.  Another important aspect is the development of Human-Centered Information Systems. The most important issue here is how to achieve synergism between man and machine. The term &#034;Human-Centered&#034; is used to emphasize the fact that although all existing information systems were designed with human users in mind, many of them are far from being user friendly. What can the scientific/engineering community do to effect a change for the better?  Information systems are ubiquitous in all human endeavors including scientific, medical, military, transportation, and consumer. Individual users use them for learning, searching for information (including data mining), doing research (including visual computing), and authoring. Multiple users (groups of users, and groups of groups of users) use them for communication and collaboration. And either single or multipl
Authentic Emotion Detection in Real%2DTime Video.
There is a growing trend toward emotional intelligence in humancomputer  interaction paradigms. In order to react appropriately to a human, the  computer would need to have some perception of the emotional state of the human.
Authentic Facial Expression Analysis.
It is argued that for the computer to be able to interact with humans, it needs to have the communication skills of humans. One of these skills is the ability to understand the emotional state of the person. The most expressive way humans display emotions is through facial expressions. In most facial expression systems and databases, the emotion data was collected by asking the subjects to perform a series of facial expressions. However, these directed or deliberate facial action tasks typically differ in appearance and timing from the authentic facial expressions induced through events in the normal environment of the subject. In this paper, we present our effort in creating an authentic facial expression database based on spontaneous emotions derived from the environment. Furthermore, we test and compare a wide range of classifiers from the machine learning literature that can be used for facial expression classification.
Toward an improved error metric.
In many computer vision algorithms, the well known Euclidean or SSD (sum of the squared differences) metric is prevalent and justified from a maximum likelihood perspective when the additive noise is Gaussian. However, Gaussian noise distribution assumption is often invalid. Previous research has found that other metrics such as double exponential metric or Cauchy metric provide better results, in accordance with the maximum likelihood approach. In this paper, we examine different error metrics and provide a theoretical approach to derive a rich set of nonlinear estimations. Our results on image databases show more robust results are obtained for noise estimation based on the proposed error metric analysis. 1.
A new analysis of the value of unlabeled data in semi%2Dsupervised learning for image retrieval.
There has been an increasing interest in using unlabeled data in semi-supervised learning for various classification problems. Previous work shows that unlabeled data can improve or degrade the classification performance depending on whether the model assumption matches the ground-truth data distribution, and also on the complexity of the classifier compared with the size of the labeled training set. In this paper, we provide a new analysis on the value of unlabeled data by considering different distributions of the labeled and unlabeled data and showing the migrating effect for semi-supervised learning. Extensive experiments have been performed in the context of image retrieval application. Our approach evaluates the value of unlabeled data from a new aspect and is aimed to provide a guideline on how unlabeled data should be used. 1.
Skin Detection A Bayesian Network Approach.
The automated detection and tracking of humans in computer vision necessitates improved modeling of the human skin appearance. In this paper we propose a Bayesian network approach for skin detection. We test several classifiers and propose a methodology for incorporating unlabeled data. We apply the semi-supervised approach to skin detection and we show that learning the structure of Bayesian network classifiers enables learning good classifiers with a small labeled set and a large unlabeled set.
Robust Error Metric Analysis for Noise Estimation in Image Indexing.

Boosting contextual information in content%2Dbased image retrieval.

Complete Performance Graphs in Probabilistic Information Retrieval.

Towards authentic emotion recognition.

Semisupervised Learning of Classifiers Theory
Automatic classification is one of the basic tasks required in any pattern recognition and human  computer interaction application. In this paper we discuss training probabilistic classifiers with labeled  and unlabeled data. We provide a new analysis that shows under what conditions unlabeled data can be  used in learning to improve classification performance. We also show that if the conditions are violated,  using unlabeled data can be detrimental to classification performance. We discuss the implications  of this analysis to a specific type of probabilistic classifiers, Bayesian networks, and propose a new  structure learning algorithm that can utilize unlabeled data to improve classification. Finally, we show  how the resulting algorithms are successfully employed in two applications related to human-computer  interaction and pattern recognition; facial expression recognition and face detection.
Proceedings of the 5th ACM SIGMM International Workshop on Multimedia Information Retrieval

Image and Video Retrieval

The State of the Art in Image and Video Retrieval.

Evaluation of Expression Recognition Techniques.
The most expressive way humans display emotions is through  facial expressions. In this work we report on several advances we have  made in building a system for classification of facial expressions from  continuous video input. We introduce and test di#erent Bayesian network  classifiers for classifying expressions from video. In particular we  use Naive-Bayes classifiers and to learn the dependencies among di#erent  facial motion features we use Tree-Augmented Naive Bayes (TAN)  classifiers. We also investigate a neural network approach. Further, we  propose an architecture of hidden Markov models (HMMs) for automatically  segmenting and recognizing human facial expression from video  sequences. We explore both person-dependent and person-independent  recognition of expressions and compare the di#erent methods.
Learning Bayesian Network Classifiers for Facial Expression Recognition using both Labeled and Unlabeled Data.
Understanding human emotions is one of the necessary skills for the computer to interact intelligently with human users. The most expressive way humans display emotions is through facial expressions. In this paper, we report on several advances we have made in building a system for classification of facial expressions from continuous video input. We use Bayesian network classifiers for classifying expressions from video. One of the motivating factor in using the Bayesian network classifiers is their ability to handle missing data, both during inference and training. In particular, we are interested in the problem of learning with both labeled and unlabeled data. We show that when using unlabeled data to learn classifiers, using correct modeling assumptions is critical for achieving improved classification performance. Motivated by this, we introduce a classification driven stochastic structure search algorithm for learning the structure of Bayesian network classifiers. We show that with moderate size labeled training sets and large amount of unlabeled data, our method can utilize unlabeled data to improve classification performance. We also provide results using the Naive Bayes (NB) and the Tree-Augmented Naive Bayes (TAN) classifiers, showing that the two can achieve good performance with labeled training sets, but perform poorly when unlabeled data are added to the training set.
Content%2Dbased indexing performance size normalized precision

Semi%2Dsupervised learning for facial expression recognition.
Automatic classification by machines is one of the basic tasks required in any pattern recognition and human computer interaction applications. In this paper, we discuss training probabilistic classifiers with labeled and unlabeled data. We provide an analysis which shows under what conditions unlabeled data can be used in learning to improve classification performance. We discuss the implications of this analysis to a specific type of probabilistic classifiers, Bayesian networks, and propose a structure learning algorithm that can utilize unlabeled data to improve classification. Finally, we show how the resulting algorithms are successfully employed in a facial expression recognition application.
Facial expression recognition from video sequences temporal and static modeling.
Human-computer intelligent interaction (HCII) is an emerging field of science aimed at providing natural ways for humans to use computers as aids. It is argued that for the computer to be able to interact with humans, it needs to have the communication skills of humans. One of these skills is the ability to understand the emotional state of the person. The most expressive way humans display emotions is through facial expressions. In this work we report on several advances we have made in building a system for classification of facial expressions from continuous video input. We introduce and test different architectures, focusing on changes in distribution assumptions and feature dependency structures. We also introduce a facial expression recognition from live video input using temporal cues. Methods for using temporal information have been extensively explored for speech recognition applications. Among these methods are template matching using dynamic programming methods and hidden Markov models (HMM). This work exploits existing methods and proposes a new architecture of HMMs for automatically segmenting and recognizing human facial expression from video sequences. The architecture performs both segmentation and recognition of the facial expressions automatically using an multi-level architecture composed of an HMM layer and a Markov model layer. We explore both person-dependent and person-independent recognition of expressions and compare the different methods.
Video retrieval and summarization.
We present a system for retrieval and summarization of continuously archived multimedia data from a home-like ubiquitous environment. Data from pressure-based floor sensors are analyzed to index video and audio from a large number of sources. Video and audio handover are implemented to retrieve continuous video streams with sound as a person is moving in the environment. Key frame extraction is proposed and several algorithms are implemented to obtain compact summaries corresponding to the activity of each person. Clustering algorithms and image analysis are used to identify actions and events. The system consists of a graphical user interface that can be used to retrieve video summaries interactively using simple queries.
Evaluation of salient point techniques.
In image retrieval, global features related to color or texture  are commonly used to describe the image content. The probleln with  this approach is that these global features cannot capture all parts of  the image having different characteristics. Therefore, local computation  of image information is necessary. By using salient points to represent  local information, more discriminative features can be computed. In this  paper we compare a wavelet-based salient point extraction algorithm  with two corner detectors using the criteria: repeatability rate and information  content. We also show that extracting color and texture information  in the locations given by onr salient points provides significantly  improved results in terms of retrieval accuracy, computational complexity,  and storage space of feature vectors as compared to global feature  approaches.
Comparing salient point detectors.
The us ofs23(&#039;H points incontent-basH retrievalallows an image index torepresq2 localproperties of the image. Clase. cornerdetectors can als beusz forthis purpos but they havedrawbacks when are applied tovarious natural images mainly becaus viss features do not need to becorners andcorners may gather ins2qH regions Inthis paper, wepresHH as3z)q2 point detectorusec wavelettranst2&#039; and we compare it with two cornerdetectors usec two criteria: repeatability rate and information content. We determine which detectorgives the besresqH) and sd2 that it st2(37) the criteria well.
Image and Video Retrieval

Challenges of Image and Video Retrieval.

Robust Shape Matching.
Many visual matching algorithms can be described in terms  of the features and the inter-feature distance or metric. The most commonly  used metric is the sum of squared di#erences (SSD), which is valid  from a maximum likelihood perspective when the real noise distribution  is Gaussian. However, we have found experimentally that the Gaussian  noise distribution assumption is often invalid. This implies that other  metrics, which have distributions closer to the real noise distribution,  should be used. In this paper we considered a shape matching application.
Object Recognition for Video Retrieval.
Recognition of objects in video can offer significant benefits  to video retrieval including automatic annotation and content based  queries based on the object characteristics. This paper describes our  preliminary work toward recognizing objects in video sequences and  gives a brief survey of the relevant research in the literature. We use  the Kalman filter to obtain segmented blobs from the video, classify the  blobs using the probability ratio test, and apply several different  temporal filtering methods, which results in sequential classification  methods over the video sequence containing the blob. Results from real  video sequences are shown.
Evaluation of Salient Point Techniques.
In image retrieval, global features related to color or texture  are commonly used to describe the image content. The probleln with  this approach is that these global features cannot capture all parts of  the image having different characteristics. Therefore, local computation  of image information is necessary. By using salient points to represent  local information, more discriminative features can be computed. In this  paper we compare a wavelet-based salient point extraction algorithm  with two corner detectors using the criteria: repeatability rate and information  content. We also show that extracting color and texture information  in the locations given by onr salient points provides significantly  improved results in terms of retrieval accuracy, computational complexity,  and storage space of feature vectors as compared to global feature  approaches.
Emotion Recognition Using a Cauchy Naive Bayes Classifier.
Recognizing human facial expression and emotion by computer is an interesting and challenging problem. In this paper we propose a method for recognizing emotions through facial expressions displayed in video sequences. We introduce the Cauchy Naive Bayes classifier which uses the Cauchy distribution as the model distribution and we provide a framework for choosing the best model distribution assumption. Our person-dependent and person-independent experiments show that the Cauchy distribution assumption typically provides better results than the Gaussian distribution assumption.
Detecting Automobiles and People for Semantic Video Retrieval.
This paper describes a method for detecting automobiles and people in streaming or archived video. Our video object tracking system is based on Kalman filter updating of an active contour over the video sequence. We use the sequential probability ratio test (SPRT) to classify the moving objects. Results are shown of a real video sequence from a busy city intersection.
Salient Points for Content%2DBased Retrieval.
In image retrieval, global features related to color or texture are commonly used to describe the image content. The problem with this approach is that these global features cannot capture all parts of the image having different characteristics. Therefore, local computation of image information is necessary. By using salient points to represent local information, more discriminative features can be computed. We show that extracting color and texture information in the locations given by our salient points provides significantly improved results as compared to global feature approaches. Moreover, our salient points lead to a more complete image representation than using corners.
Extended Performance Graphs for Cluster Retrieval.
Performance evaluations in Probabilistic Information Retrieval are often presented as Precision-Recall or PrecisionScope graphs avoiding the otherwise dominating effect of the embedding irrelevant fraction. However precision and recall values as such offer an incomplete overview of the information retrieval system under study: information about system parameters like generality (the embedding of the relevant fraction), random performance and the effect of varying the scope is badly missed.
Comparing Salient Point Detectors.
The us ofs23(&#039;H points incontent-basH retrievalallows an image index torepresq2 localproperties of the image. Clase. cornerdetectors can als beusz forthis purpos but they havedrawbacks when are applied tovarious natural images mainly becaus viss features do not need to becorners andcorners may gather ins2qH regions Inthis paper, wepresHH as3z)q2 point detectorusec wavelettranst2&#039; and we compare it with two cornerdetectors usec two criteria: repeatability rate and information content. We determine which detectorgives the besresqH) and sd2 that it st2(37) the criteria well.
Content%2Dbased image retrieval using wavelet%2Dbased salient points.
Content-based Image Retrieval (CBIR) has become one of the most active research areas in the past few years. Most of the attention from the research has been focused on indexing techniques based on global feature distributions. However, these global distributions have limited discriminating power because they are unable to capture local image information. Applying global Gabor texture features greatly improve the retrieval accuracy. But they are computationally complex. In this paper, we present a wavelet-based salient point extraction algorithm. We show that extracting the color and texture information in the locations given by these points provides significantly improved results in terms of retrieval accuracy, computational complexity and storage space of feature vectors as compared to the global feature approaches.
Image retrieval using wavelet%2Dbased salient points.
Content-based Image Retrieval (CBIR) has become one of the most active research areas in the past few years. Most of the attention from the research has been focused on indexing techniques based on global feature distributions. However, these global distributions have limited discriminating power because they are unable to capture local image information. Applying global Gabor texture features greatly improve the retrieval accuracy. But they are computationally complex. In this paper, we present a wavelet-based salient point extraction algorithm. We show that extracting the color and texture information in the locations given by these points provides significantly improved results in terms of retrieval accuracy, computational complexity and storage space of feature vectors as compared to the global feature approaches.
Color%2Dbased retrieval.
In content based retrieval, color indexing is one of the most prevalent retrieval methods. Two key problems in color indexing are (1) determination of the color space and (2) finding the best distance measure. Most of the attention from the research literature has been focussed on the color model with little or no consideration of the noise models. By focusing on the noise model, we showed how to find a distance measure which is optimal in the sense that it maximizes the similarity probability. Our results include experiments in color based retrieval and object recognition. Furthermore, we implemented and tested several promising algorithms from the research literature as benchmarks.  1. INTRODUCTION  As the world enters the digital age, visual media is becoming prevalent and easily accessible. Factors such as the explosive growth of the World Wide Web, terabyte disk servers, and the digital versatile disk, reveal the growing amount of visual media which is available to society. With t...
Maximum Likelihood Stereo Matching.
In the research literature, maximum likelihood principles were applied to stereo matching by altering the stereo pair so that the difference would have a Gaussian distribution. Here in this paper we present a novel method of applying maximum likelihood to stereo matching. In our approach, we measure the real noise distribution from a training set, and then construct a new metric which we denote the maximum likelihood metric for comparing the stereo pair. The maximum likelihood metric is optimal in the sense that it maximizes the probability of similarity. In our experiments and discussion, we compared the maximum likelihood metric to other promising algorithms from the research literature using international stereo data sets. Furthermore, we showed that the algorithms from the research literature could be improved by using the maximum likelihood metric instead of the sum of squared differences.  1. Introduction  The common assumption is that the real noise distribution should fit eithe...
Improving Visual Matching.
Many visual matching algorithms can be described in terms of the features and the inter-feature distance or metric. The most commonly used metric is the sum of squared differences (SSD), which is valid from a maximum likelihood perspective when the real noise distribution is Gaussian. Based on real noise distributions measured from international test sets, we have found experimentally that the Gaussian noise distribution assumption is often invalid. This implies that other metrics, which have distributions closer to the real noise distribution, should be used. In this paper we considered two different visual matching applications: content-based retrieval in image databases and stereo matching. Towards broadening the results, we also implemented several sophisticated algorithms from the research literature. In each algorithm we compared the efficacy of the SSD metric, the SAD (sum of the absolute differences) metric, the Cauchy metric, and the Kullback relative information. Furthermore,...
Visual Websearching Using Iconic Queries.
Imagine a professional Web site designer who constantly has to come up with innovative looks for the client&#039;s homepages? How does he find the new content? The major search engines such as Hotbot (http://www.hotbot.com) allow us to find text on the Web, but typically have few or no capabilities for finding visual media. In this article we discuss methods for finding visual media on the WWW. The emphasis is on iconic queries which are essentially drag-and-drop visual concepts or simple semantics. We describe our method for finding static feature sets and then describe a novel method called active feature sets, which chooses a feature set based on the context in the image.
Wavelet%2DBased Salient Points for Image Retrieval.
The use of interest points in content-based image retrieval allows image index to represent local properties of the image. Classic corner detectors can be used for this purpose. However, they have drawbacks when applied to various natural images for image retrieval, because visual features need not be corners and corners may gather in small regions.
Maximum Likelihood Stereo Matching.
In the research literature, maximum likelihood principles were applied to stereo matching by altering the stereo pair so that the difference would have a Gaussian distribution. Here in this paper we present a novel method of applying maximum likelihood to stereo matching. In our approach, we measure the real noise distribution from a training set, and then construct a new metric which we denote the maximum likelihood metric for comparing the stereo pair. The maximum likelihood metric is optimal in the sense that it maximizes the probability of similarity. In our experiments and discussion, we compared the maximum likelihood metric to other promising algorithms from the research literature using international stereo data sets. Furthermore, we showed that the algorithms from the research literature could be improved by using the maximum likelihood metric instead of the sum of squared differences.  1. Introduction  The common assumption is that the real noise distribution should fit eithe...
Wavelet Based Texture Classification.
Textures are one of the basic features in visual searching and computational vision. In the literature, most of the attention has been focussed on the texture features with minimal consideration of the noise models. In this paper we investigated the problem of texture classification from a maximum likelihood perspective. We took into account the texture model, the noise distribution, and the inter-dependence of the texture features. Our investigation showed that the real noise distribution is closer to an Exponential than a Gaussian distribution, and that the L 1 metric has a better retrieval rate than L 2 . We also proposed the Cauchy metric as an alternative for both the L 1 and L 2 metrics. Furthermore, we provided a direct method for deriving an optimal distortion measure from the real noise distribution, which experimentally provides consistently improved results over the other metrics. We conclude with results and discussions on an international texture database.  1. Introduction...
Color Based Retrieval and Recognition.
In content based retrieval, color indexing is one of the most prevalent retrieval methods. Two key problems in color indexing are (1) determination of the color space and (2) finding the best distance measure. Most of the attention from the research literature has been focussed on the color model with little or no consideration of the noise models. By focusing on the noise model, we showed how to find a distance measure which is optimal in the sense that it maximizes the similarity probability. Our results include experiments in color based retrieval and object recognition. Furthermore, we implemented and tested several promising algorithms from the research literature as benchmarks.  1. INTRODUCTION  As the world enters the digital age, visual media is becoming prevalent and easily accessible. Factors such as the explosive growth of the World Wide Web, terabyte disk servers, and the digital versatile disk, reveal the growing amount of visual media which is available to society. With t...
Wavelet%2DBased Salient Points Applications to Image Retrieval Using Color and Texture Features.
In image retrieval, global features related to color or texture are  commonly used to describe the image. The use of interest points in contentbased  image retrieval allows image index to represent local properties of images.
A Ground%2DTruth Training Set for Hierarchical Clustering in Content%2DBased Image Retrieval.
Progress in Content-Based Image Retrieval (CBIR) is hampered by the absence of well-documented and validated test-sets that provide ground-truth for the performance evaluation of image indexing, retrieval and clustering tasks. For quick access to large (tenthousands or millions of images) digital image collections a hierarchically structured indexing or browsing mechanism based on clusters of similar images at various coarse to fine levels is highly wanted. The Leiden 19th-Century Portrait Database (LCPD), that consists of over 16,000 scanned studio portraits (so-called Cartes de Visite CdV), happens to have a clearly delineated set of clusters in the studio logo backside images. Clusters of similar or semantically identical logos can also be formed on a number of levels that show a clear hierarchy. The Leiden Imaging and Multimedia Group is constructing a CD-ROM with a well-documented set of studio portraits and logos that can serve as ground-truth for feature performance evaluation in domains beside color-indexing. Its grey-level image lay-out characteristics are also described by various precalculated feature vector sets. For both portraits (near copy pairs) and studio logos (clusters of identical logos) test-sets will be provided and described at various clustering levels. The statistically significant number of test-set images embedded in a realistically large environment of narrow-domain images are presented to the CBIR community to enable selection of more optimal indexing and retrieval approaches as part of an internationally defined test-set that comprises test-sets specifically designed for color-, texture- and shape retrieval evaluation.
Toward Improved Ranking Metrics.
AbstractÐIn many computer vision algorithms, a metric or similarity measure is used to determine the distance between two features. The Euclidean or SSD (sum of the squared differences) metric is prevalent and justified from a maximum likelihood perspective when the additive noise distribution is Gaussian. Based on real noise distributions measured from international test sets, we have found that the Gaussian noise distribution assumption is often invalid. This implies that other metrics, which have distributions closer to the real noise distribution, should be used. In this paper, we consider three different applications: content-based retrieval in image databases, stereo matching, and motion tracking. In each of them, we experiment with different modeling functions for the noise distribution and compute the accuracy of the methods using the corresponding distance measures. In our experiments, we compared the SSD metric, the SAD (sum of the absolute differences) metric, the Cauchy metric, and the Kullback relative information. For several algorithms from the research literature which used the SSD or SAD, we showed that greater accuracy could be obtained by using the Cauchy metric instead. Index TermsÐMaximum likelihood, ranking metrics, content-based retrieval, color indexing, stereo matching, motion tracking. 1
Robust color indexing.
In content based image retrieval, color indexing is one of the most prevalent retrieval methods. In literature, most of the attention has been focussed on the color model with little or no consideration of the noise models. In this paper we investigate the problem of color indexing from a maximum likelihood perspective. We take into account the color model, the noise distribution, and the quantization of the color features. Furthermore, from the real noise distribution we derive a distortion measure, which consistently provides improved accuracy. Our investigation concludes with results on a real stock photography database, consisting of 11,000 color images.  1 INTRODUCTION  Of the visual media retrieval methods, color indexing is one of the dominant methods because it has been shown to be effective in both the academic and commercial arenas. In color indexing, histogram methods are often used because they are feasible in terms of memory usage and provide sufficient accuracy. The histo...
Multi%2Dscale sub%2Dimage search.
If an image should be retrieved by its subregions from a large image database, an immense number of possible queries will appear. Therefore, the index which encodes the spatial information of an image, should make only few assumptions about possible queries. In addition, this index has to consider different scales of objects in the image. In this paper, we propose a novel approach using a hierarchical index encoding image regions, gained by a fixed partition. The suggested index uses color features and is easy to implement. The index is tested on a database with more than 12,000 images.  1 INTRODUCTION  While most image retrieval systems retrieve images based on overall image comparison, users are typically interested in &#034;object searching&#034;, where they can specify an &#034;interesting&#034; subregion (usually an interesting object) of an image, as a query. The system should then retrieve images containing this subregion (according to human perception) from a database. This task called sub-image q...
Adapting k%2Dd Trees to Visual Retrieval.
. The most frequently occurring problem in image retrieval is  find-the-similar-image, which in general is finding the nearest neighbor.  From the literature, it is well known that k-d trees are efficient methods  of finding nearest neighbors in high dimensional spaces. In this paper  we survey the relevant k-d tree literature, and adapt the most promising  solution to the problem of image retrieval by finding the best parameters  for the bucket size and threshold. We also test the system on the Corel  Studio photo database of 18,724 images and measure the user response  times and retrieval accuracy.  1 Introduction  Searching for multimedia is one of the outstanding problems for the World Wide Web and for companies which have networked databases. For example, Philips Media uses tera-byte databases which are derived from ownership of trademarks, video, images, and audio. The real world usage of the multimedia search programs is to support and to enable the re-use and trading of assets....
Generation of the HDL%2DA%2Dmodel of a micromembrane from its finite%2Delement%2Ddescription.

